[1] TRUEWeb Scraping
Day 21
Prof Emily Kurtz
Carleton College
Stat 220 - Spring 2026
Ways to access data from the web:
Web APIs (application programming interface): website offers a set of structured http requests that return JSON or XML files.
Screen scraping:
extract data from source code of website, with html parser (easy) or regular expression matching (less easy).
API vs Screen Scraping
API:
- Designed to be accessed by computers
- Often need to sign up for a key
- Structured set of requests, need to dig in to documentation to figure out how to access the data you need
- Often more natural structure for tidying
Screen scraping:
- Designed to be read by humans
- Can be restricted or rate-limited in terms of service
- Need to dig in to source code of the web page to figure out how to access the data that you need
Check the terms of use/service first!
Can you query this webpage?
Are there restrictions on the use of the data?
How many requests can you make per minute?
…and more…
Checking for permission to scrape through R
Use robotstxt::paths_allowed() to see if you can scrape the web page.
You can scrape Zillow
What websites have data about you? Think of 1-2 and see if scraping is allowed on those sites.
Hypertext Markup Language
Lots of data on the web is still available as HTML
It is structured (hierarchical / tree based), but it’s often not available in a form useful for analysis (flat / tidy).
HTML tags
HTML uses tags to describe different aspects of document content
| Tag | Example |
|---|---|
| heading | <h1>My Title</h1> |
| paragraph | <p>A paragraph of content...</p> |
| table | <table> ... </table> |
| anchor (with attribute) | <a href="http://www.mysite.net">click here for link</a> |
{rvest}
Core rvest functions
| Function | Description |
|---|---|
read_html |
Read HTML data from a url or character string |
html_element |
Select a specified element from HTML document |
html_elements |
Select specified elements from HTML document |
html_table |
Parse an HTML table into a data frame |
html_text |
Extract tag pairs’ content |
html_name |
Extract tags’ names |
html_attrs |
Extract all of each tag’s attributes |
html_attr |
Extract tags’ attribute value by name |
Example: box office mojo
https://www.boxofficemojo.com/year/2024/
Take a look at the web page and the html source code
Chrome or Firefox: right click -> View page source
right click -> Inspect will also help and highlight what part of site corresponds to what html code
Look for the
"table"div ID or tag
Read HTML into R
{html_document}
<html class="a-no-js" data-19ax5a9jf="dingo">
[1] <head>\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8 ...
[2] <body id="body" class="mojo-page-id-yld a-m-us a-aui_72554-c a-aui_templa ...List of 2
$ node:<externalptr>
$ doc :<externalptr>
- attr(*, "class")= chr [1:2] "xml_document" "xml_node"HTML elements
There are over 100 HTML elements:
- Every HTML page must be in an
<html>element, and it must have two children:<head>and<body> - Block tags like
<h1>,<p>,<ol>form the structure of the page - Inline tags like
<b>,<i>, and<a>format text inside block tags
We’ll often work with tables. HTML tables are composed of four main elements <table>, <tr> (table row), <th> (table heading), and <td> (table data).
Extract tables
Use html_element() or html_elements() to extract pieces out of HTML documents
html_element() vs html_elements()
html_elements()returns all matching elements beneath any of the inputs, flattening results into a new node sethtml_element()always returns a vector the same length as the input, using a “missing” element where needed.
Typically, we’ll use html_elements to get the overall structure for our data, followed by something else (sometimes html_element, sometimes html_table) to access what we need
Check that it’s the right table
It looks promising!
{xml_nodeset (1)}
[1] <table class="a-bordered a-horizontal-stripes a-size-base a-span12 mojo-b ...But we don’t have a data frame yet…
{html_node}
<table class="a-bordered a-horizontal-stripes a-size-base a-span12 mojo-body-table mojo-table-annotated mojo-body-table-compact">
[1] <tr>\n<th class="a-text-right mojo-field-type-rank mojo-sort-column mojo ...
[2] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[3] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[4] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[5] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[6] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[7] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[8] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[9] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[10] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[11] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[12] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[13] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[14] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[15] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[16] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[17] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[18] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[19] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
[20] <tr>\n<td class="a-text-right mojo-header-column mojo-truncate mojo-fiel ...
...Parse a table into a data frame/tibble
Rows: 200
Columns: 11
$ Rank <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, …
$ Release <chr> "Inside Out 2", "Deadpool & Wolverine", "Wicked", "Moan…
$ Genre <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ Budget <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ `Running Time` <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ Gross <chr> "$652,980,194", "$636,745,858", "$432,943,285", "$404,0…
$ Theaters <chr> "4,440", "4,330", "3,888", "4,200", "4,449", "4,575", "…
$ `Total Gross` <chr> "$652,980,194", "$636,745,858", "$474,983,975", "$460,4…
$ `Release Date` <chr> "Jun 14", "Jul 26", "Nov 22", "Nov 27", "Jul 3", "Sep 6…
$ Distributor <chr> "Walt Disney Studios Motion Pictures", "Walt Disney Stu…
$ Estimated <chr> "false", "false", "false", "false", "false", "false", "…Scrape then wrangle
Data aren’t ready for analysis, too many character columns!
Rows: 200
Columns: 11
$ Rank <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, …
$ Release <chr> "Inside Out 2", "Deadpool & Wolverine", "Wicked", "Moan…
$ Genre <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ Budget <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ `Running Time` <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ Gross <chr> "$652,980,194", "$636,745,858", "$432,943,285", "$404,0…
$ Theaters <chr> "4,440", "4,330", "3,888", "4,200", "4,449", "4,575", "…
$ `Total Gross` <chr> "$652,980,194", "$636,745,858", "$474,983,975", "$460,4…
$ `Release Date` <chr> "Jun 14", "Jul 26", "Nov 22", "Nov 27", "Jul 3", "Sep 6…
$ Distributor <chr> "Walt Disney Studios Motion Pictures", "Walt Disney Stu…
$ Estimated <chr> "false", "false", "false", "false", "false", "false", "…Scrape then wrangle
Rows: 200
Columns: 12
$ Rank <int> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, …
$ Release <chr> "Inside Out 2", "Deadpool & Wolverine", "Wicked", "Moan…
$ Genre <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ Budget <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ `Running Time` <chr> "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", "-", …
$ Gross <dbl> 652980194, 636745858, 432943285, 404017489, 361004205, …
$ Theaters <dbl> 4440, 4330, 3888, 4200, 4449, 4575, 4074, 4170, 3948, 4…
$ `Total Gross` <dbl> 652980194, 636745858, 474983975, 460405297, 361004205, …
$ Month <chr> "Jun", "Jul", "Nov", "Nov", "Jul", "Sep", "Mar", "Jul",…
$ Day <chr> "14", "26", "22", "27", "3", "6", "1", "19", "29", "8",…
$ Distributor <chr> "Walt Disney Studios Motion Pictures", "Walt Disney Stu…
$ Estimated <chr> "false", "false", "false", "false", "false", "false", "…Scraped data will almost always need wrangling/cleaning
- Are numeric columns numeric?
- Are date columns dates?
- Are factor and string columns treated correctly?
Data aren’t always stored as tables
https://www.carleton.edu/catalog/current/search/?subject=STAT&term=26SP
View the page source to try to find the html elements where this data is located (e.g. ‘h1’, ‘p’, ‘table’)
- Course number
- Course title
- Course description
- Course meetings
- Faculty
- Course meetings
{xml_nodeset (19)}
[1] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[2] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[3] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[4] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[5] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[6] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[7] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[8] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[9] <h3 class="courseSearchResultsHeading relatedCourses" id="relatedCourses ...
[10] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[11] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[12] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[13] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[14] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[15] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[16] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[17] <h3 class="courseTitleBar">\n <span class="courseNumber" data ...
[18] <h3>Liberal Arts Requirements</h3>
[19] <h3>Other Course Tags</h3> [1] "\n STAT 120\n Introduction to Statistics\n \n 6 credits\n \n "
[2] "\n STAT 220\n Introduction to Data Science\n \n 6 credits\n \n "
[3] "\n STAT 230\n Applied Regression Analysis\n \n 6 credits\n \n "
[4] "\n STAT 250\n Introduction to Statistical Inference\n \n 6 credits\n \n "
[5] "\n STAT 285\n Statistical Consulting\n \n 2 credits\n \n "
[6] "\n STAT 297\n Assessment and Communication of External Statistical Activity\n \n 1 credits\n \n "
[7] "\n STAT 330\n Advanced Statistical Modeling\n \n 6 credits\n \n "
[8] "\n STAT 400\n Integrative Exercise\n \n 3 credits\n \n "
[9] "Related Courses"
[10] "\n CS 111\n Introduction to Computer Science\n \n 6 credits\n \n "
[11] "\n CS 314*\n Data Visualization (*=Junior Seminar)\n \n 6 credits\n \n "
[12] "\n CS 362\n Computational Biology\n \n 6 credits\n \n "
[13] "\n MATH 120\n Calculus 2\n \n 6 credits\n \n "
[14] "\n MATH 134\n Linear Algebra with Applications\n \n 6 credits\n \n "
[15] "\n MATH 210\n Calculus 3\n \n 6 credits\n \n "
[16] "\n MATH 232\n Linear Algebra\n \n 6 credits\n \n "
[17] "\n MATH 271\n Optimization\n \n 6 credits\n \n "
[18] "Liberal Arts Requirements"
[19] "Other Course Tags" [1] "STAT 120 Introduction to Statistics 6 credits"
[2] "STAT 220 Introduction to Data Science 6 credits"
[3] "STAT 230 Applied Regression Analysis 6 credits"
[4] "STAT 250 Introduction to Statistical Inference 6 credits"
[5] "STAT 285 Statistical Consulting 2 credits"
[6] "STAT 297 Assessment and Communication of External Statistical Activity 1 credits"
[7] "STAT 330 Advanced Statistical Modeling 6 credits"
[8] "STAT 400 Integrative Exercise 3 credits"
[9] "Related Courses"
[10] "CS 111 Introduction to Computer Science 6 credits"
[11] "CS 314* Data Visualization (*=Junior Seminar) 6 credits"
[12] "CS 362 Computational Biology 6 credits"
[13] "MATH 120 Calculus 2 6 credits"
[14] "MATH 134 Linear Algebra with Applications 6 credits"
[15] "MATH 210 Calculus 3 6 credits"
[16] "MATH 232 Linear Algebra 6 credits"
[17] "MATH 271 Optimization 6 credits"
[18] "Liberal Arts Requirements"
[19] "Other Course Tags" CSS selectors
Selecting courseNumber class
Course numbers are between <span class="courseNumber"> ... </span> tags
These tags can be selected using . followed by the name of the class
{xml_nodeset (16)}
[1] <span class="courseNumber" data-terms="26/SP">STAT 120</span>
[2] <span class="courseNumber" data-terms="26/SP">STAT 220</span>
[3] <span class="courseNumber" data-terms="26/SP">STAT 230</span>
[4] <span class="courseNumber" data-terms="26/SP">STAT 250</span>
[5] <span class="courseNumber" data-terms="26/SP">STAT 285</span>
[6] <span class="courseNumber" data-terms="26/SP">STAT 297</span>
[7] <span class="courseNumber" data-terms="26/SP">STAT 330</span>
[8] <span class="courseNumber" data-terms="26/SP">STAT 400</span>
[9] <span class="courseNumber" data-terms="26/SP">CS 111</span>
[10] <span class="courseNumber" data-terms="26/SP">CS 314*</span>
[11] <span class="courseNumber" data-terms="26/SP">CS 362</span>
[12] <span class="courseNumber" data-terms="26/SP">MATH 120</span>
[13] <span class="courseNumber" data-terms="26/SP">MATH 134</span>
[14] <span class="courseNumber" data-terms="26/SP">MATH 210</span>
[15] <span class="courseNumber" data-terms="26/SP">MATH 232</span>
[16] <span class="courseNumber" data-terms="26/SP">MATH 271</span>Scraping courseNumbers
Scraping credits
[1] "\n 6 credits\n "
[2] "\n 6 credits\n "
[3] "\n 6 credits\n "
[4] "\n 6 credits\n "
[5] "\n 2 credits\n "
[6] "\n 1 credits\n "
[7] "\n 6 credits\n "
[8] "\n 3 credits\n "
[9] "\n 6 credits\n "
[10] "\n 6 credits\n "
[11] "\n 6 credits\n "
[12] "\n 6 credits\n "
[13] "\n 6 credits\n "
[14] "\n 6 credits\n "
[15] "\n 6 credits\n "
[16] "\n 6 credits\n "Scraping credits
stat_spring2026 <- tibble(
course = listings %>% html_elements(".courseNumber") %>% html_text(),
title = listings %>% html_elements(".courseTitle") %>% html_text(),
credits = listings %>% html_elements(".credits") %>% html_text() %>% str_squish(),
description = listings %>% html_elements(".courseDetailWrapper") %>% html_text() %>% str_squish()
)
stat_spring2026# A tibble: 16 × 4
course title credits description
<chr> <chr> <chr> <chr>
1 STAT 120 Introduction to Statistics 6 cred… "Introduct…
2 STAT 220 Introduction to Data Science 6 cred… "This cour…
3 STAT 230 Applied Regression Analysis 6 cred… "A second …
4 STAT 250 Introduction to Statistical Inference 6 cred… "Introduct…
5 STAT 285 Statistical Consulting 2 cred… "Students …
6 STAT 297 Assessment and Communication of External Statis… 1 cred… "An indepe…
7 STAT 330 Advanced Statistical Modeling 6 cred… "Topics in…
8 STAT 400 Integrative Exercise 3 cred… "A supervi…
9 CS 111 Introduction to Computer Science 6 cred… "This cour…
10 CS 314* Data Visualization (*=Junior Seminar) 6 cred… "Though th…
11 CS 362 Computational Biology 6 cred… "Recent ad…
12 MATH 120 Calculus 2 6 cred… "Inverse f…
13 MATH 134 Linear Algebra with Applications 6 cred… "Linear al…
14 MATH 210 Calculus 3 6 cred… "Vectors, …
15 MATH 232 Linear Algebra 6 cred… "Linear al…
16 MATH 271 Optimization 6 cred… "Optimizat…What about sections?
[1] "STAT 120.01 Spring 2026" "STAT 120.02 Spring 2026"
[3] "STAT 120.03 Spring 2026" "STAT 120.04 Spring 2026"
[5] "STAT 220.01 Spring 2026" "STAT 230.01 Spring 2026"
[7] "STAT 230.02 Spring 2026" "STAT 250.01 Spring 2026"
[9] "STAT 285.01 Spring 2026" "STAT 297.01 Spring 2026"
[11] "STAT 330.01 Spring 2026" "STAT 400.01 Spring 2026"
[13] "CS 111.01 Spring 2026" "CS 111.02 Spring 2026"
[15] "CS 314*.01 Spring 2026" "CS 362.01 Spring 2026"
[17] "MATH 120.01 Spring 2026" "MATH 134.01 Spring 2026"
[19] "MATH 210.01 Spring 2026" "MATH 232.01 Spring 2026"
[21] "MATH 232.02 Spring 2026" "MATH 271.01 Spring 2026"Sometimes, we can’t get around regex :(
[1] "STAT 120.01 Spring 2026 Faculty:Emily Kurtz 🏫 👤 Size:32 M, WCMC 102 9:50am-11:00am FCMC 102 9:40am-10:40am"
[2] "STAT 120.02 Spring 2026 Faculty:Amanda Luby 🏫 👤 Size:32 M, WCMC 102 11:10am-12:20pm FCMC 102 12:00pm-1:00pm Not open to students who have already received credit for Psychology 200/201, Sociology/Anthropology 239 or Statistics 250 Sophomore Priority"
[3] "STAT 120.03 Spring 2026 Faculty:Andy Poppick 🏫 👤 Size:32 M, WCMC 306 12:30pm-1:40pm FCMC 306 1:10pm-2:10pm"
[4] "STAT 120.04 Spring 2026 Faculty:Adam Loy 🏫 👤 Size:32 M, WCMC 102 1:50pm-3:00pm FCMC 102 2:20pm-3:20pm Not open to students who have already received credit for Psychology 200/201, Sociology/Anthropology 239 or Statistics 250 Sophomore Priority"
[5] "STAT 220.01 Spring 2026 Faculty:Emily Kurtz 🏫 👤 Size:30 M, WCMC 102 12:30pm-1:40pm FCMC 102 1:10pm-2:10pm"
[6] "STAT 230.01 Spring 2026 Faculty:Adam Loy 🏫 👤 Size:28 M, WCMC 306 11:10am-12:20pm FCMC 306 12:00pm-1:00pm"
[7] "STAT 230.02 Spring 2026 Faculty:Amanda Luby 🏫 👤 Size:28 M, WCMC 306 1:50pm-3:00pm FCMC 306 2:20pm-3:20pm Sophomore Priority"
[8] "STAT 250.01 Spring 2026 Faculty:Andy Poppick 🏫 👤 Size:28 M, WCMC 306 9:50am-11:00am FCMC 306 9:40am-10:40am"
[9] "STAT 285.01 Spring 2026 Faculty:Andy Poppick 🏫 👤 Grading:S/CR/NC TCMC 304 10:10am-11:55am All interested students are encouraged to add to the waitlist and the instructor will reach out after registration. This course is repeatable, but if the instructor cannot admit every student on the waitlist, priority will be given first to Statistics majors who have not previously taken the course and then to other students who have not taken the course. Waitlist Only"
[10] "STAT 297.01 Spring 2026 Faculty:Katie St. Clair 🏫 👤 · Rafe Jones 🏫 👤 Grading:S/CR/NC"
[11] "STAT 330.01 Spring 2026 Faculty:Katie St. Clair 🏫 👤 Size:20 M, WCMC 210 9:50am-11:00am FCMC 210 9:40am-10:40am"
[12] "STAT 400.01 Spring 2026 Faculty:Amanda Luby 🏫 👤 Size:9 Grading:S/NC This section is for the STAT Comps Group Project: STAT 399 (6 credits), then STAT 400 (3 credits)."
[13] "CS 111.01 Spring 2026 Faculty:Anna Meyer 🏫 👤 Size:48 M, WOlin 310 9:50am-11:00am FOlin 310 9:40am-10:40am Sophomore Priority"
[14] "CS 111.02 Spring 2026 Faculty:Jean Salac 🏫 👤 Size:48 M, WOlin 310 12:30pm-1:40pm FOlin 310 1:10pm-2:10pm Sophomore Priority"
[15] "CS 314*.01 Spring 2026 Faculty:Eric Alexander 🏫 👤 Size:16 M, WAnderson Hall 223 12:30pm-1:40pm FAnderson Hall 223 1:10pm-2:10pm 16 seats held for CS Match until the day after junior priority registration."
[16] "CS 362.01 Spring 2026 Faculty:Layla Oesper 🏫 👤 Size:28 M, WAnderson Hall 323 11:10am-12:20pm FAnderson Hall 323 12:00pm-1:00pm 16 seats held for CS Match until the day after sophomore only priority registration."
[17] "MATH 120.01 Spring 2026 Faculty:Deewang Bhamidipati 🏫 👤 Size:30 M, WCMC 206 9:50am-11:00am FCMC 206 9:40am-10:40am"
[18] "MATH 134.01 Spring 2026 Faculty:Kate Meyer 🏫 👤 Size:30 M, WCMC 209 11:10am-12:20pm FCMC 209 12:00pm-1:00pm"
[19] "MATH 210.01 Spring 2026 Faculty:Deewang Bhamidipati 🏫 👤 Size:30 M, WCMC 209 12:30pm-1:40pm FCMC 209 1:10pm-2:10pm"
[20] "MATH 232.01 Spring 2026 Faculty:Corey Brooke 🏫 👤 Size:30 M, WCMC 206 11:10am-12:20pm FCMC 206 12:00pm-1:00pm"
[21] "MATH 232.02 Spring 2026 Faculty:Corey Brooke 🏫 👤 Size:30 M, WCMC 206 1:50pm-3:00pm FCMC 206 2:20pm-3:20pm This course is not open to students who have received credit for MATH 134. Sophomore Priority"
[22] "MATH 271.01 Spring 2026 Faculty:Joseph Johnson 🏫 👤 Size:25 M, WCMC 206 12:30pm-1:40pm FCMC 206 1:10pm-2:10pm" selectorGadget
Open source tool that eases CSS selector generation and discovery
Easiest to use with the Chrome Extension
Find out more on the SelectorGadget vignette
selectorGadget
- Click on the app logo next to the search bar
- A box will open in the bottom right of the website
- Click on a page element (it will turn green), SelectorGadget will generate a minimal CSS selector for that element, and will highlight (yellow) everything that is matched by the selector
- Click on a highlighted element to remove it from the selector (red), or click on an unhighlighted element to add it to the selector
Try it
Use the SelectorGadget to explore http://www.imdb.com/chart/top
What should the columns of our target dataset be? Do they correspond to any specific css selectors?
Extract title
With local version of website
[1] "1. The Shawshank Redemption"
[2] "2. The Godfather"
[3] "3. The Dark Knight"
[4] "4. The Godfather Part II"
[5] "5. 12 Angry Men"
[6] "6. The Lord of the Rings: The Return of the King"
[7] "7. Schindler's List"
[8] "8. Pulp Fiction"
[9] "9. The Lord of the Rings: The Fellowship of the Ring"
[10] "10. The Good, the Bad and the Ugly"
[11] "11. Forrest Gump"
[12] "12. The Lord of the Rings: The Two Towers"
[13] "13. Fight Club"
[14] "14. Inception"
[15] "15. Star Wars: Episode V - The Empire Strikes Back"
[16] "16. The Matrix"
[17] "17. Goodfellas"
[18] "18. One Flew Over the Cuckoo's Nest"
[19] "19. Interstellar"
[20] "20. Se7en"
[21] "21. It's a Wonderful Life"
[22] "22. Seven Samurai"
[23] "23. The Silence of the Lambs"
[24] "24. Saving Private Ryan"
[25] "25. City of God"
[26] "26. The Green Mile"
[27] "27. Life Is Beautiful"
[28] "28. Terminator 2: Judgment Day"
[29] "29. Star Wars: Episode IV - A New Hope"
[30] "30. Back to the Future"
[31] "31. Spirited Away"
[32] "32. The Pianist"
[33] "33. Gladiator"
[34] "34. Parasite"
[35] "35. Psycho"
[36] "36. The Lion King"
[37] "37. Grave of the Fireflies"
[38] "38. The Departed"
[39] "39. Whiplash"
[40] "40. Harakiri"
[41] "41. American History X"
[42] "42. The Prestige"
[43] "43. Léon: The Professional"
[44] "44. Spider-Man: Across the Spider-Verse"
[45] "45. Casablanca"
[46] "46. The Usual Suspects"
[47] "47. The Intouchables"
[48] "48. Cinema Paradiso"
[49] "49. Modern Times"
[50] "50. Alien"
[51] "51. Rear Window"
[52] "52. Once Upon a Time in the West"
[53] "53. Django Unchained"
[54] "54. City Lights"
[55] "55. Dune: Part Two"
[56] "56. Apocalypse Now"
[57] "57. Memento"
[58] "58. WALL·E"
[59] "59. Raiders of the Lost Ark"
[60] "60. The Lives of Others"
[61] "61. Avengers: Infinity War"
[62] "62. Sunset Boulevard"
[63] "63. Spider-Man: Into the Spider-Verse"
[64] "64. Paths of Glory"
[65] "65. Witness for the Prosecution"
[66] "66. The Shining"
[67] "67. The Great Dictator"
[68] "68. 12th Fail"
[69] "69. Aliens"
[70] "70. Inglourious Basterds"
[71] "71. The Dark Knight Rises"
[72] "72. Coco"
[73] "73. Amadeus"
[74] "74. Toy Story"
[75] "75. Avengers: Endgame"
[76] "76. Oldboy"
[77] "77. Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb"
[78] "78. Good Will Hunting"
[79] "79. American Beauty"
[80] "80. Das Boot"
[81] "81. Braveheart"
[82] "82. Princess Mononoke"
[83] "83. Your Name."
[84] "84. High and Low"
[85] "85. 3 Idiots"
[86] "86. Joker"
[87] "87. Once Upon a Time in America"
[88] "88. Capernaum"
[89] "89. Singin' in the Rain"
[90] "90. Come and See"
[91] "91. Requiem for a Dream"
[92] "92. Toy Story 3"
[93] "93. Star Wars: Episode VI - Return of the Jedi"
[94] "94. The Hunt"
[95] "95. Eternal Sunshine of the Spotless Mind"
[96] "96. Ikiru"
[97] "97. 2001: A Space Odyssey"
[98] "98. Reservoir Dogs"
[99] "99. The Apartment"
[100] "100. Lawrence of Arabia"
[101] "101. Incendies"
[102] "102. Scarface"
[103] "103. Double Indemnity"
[104] "104. North by Northwest"
[105] "105. Heat"
[106] "106. Citizen Kane"
[107] "107. M"
[108] "108. Up"
[109] "109. Full Metal Jacket"
[110] "110. Vertigo"
[111] "111. Amélie"
[112] "112. A Clockwork Orange"
[113] "113. Oppenheimer"
[114] "114. To Kill a Mockingbird"
[115] "115. A Separation"
[116] "116. Die Hard"
[117] "117. The Sting"
[118] "118. Like Stars on Earth"
[119] "119. Indiana Jones and the Last Crusade"
[120] "120. Metropolis"
[121] "121. I'm Still Here"
[122] "122. Snatch"
[123] "123. 1917"
[124] "124. L.A. Confidential"
[125] "125. Bicycle Thieves"
[126] "126. Downfall"
[127] "127. Dangal"
[128] "128. Taxi Driver"
[129] "129. Hamilton"
[130] "130. The Wolf of Wall Street"
[131] "131. Batman Begins"
[132] "132. Green Book"
[133] "133. For a Few Dollars More"
[134] "134. Some Like It Hot"
[135] "135. The Truman Show"
[136] "136. Judgment at Nuremberg"
[137] "137. The Kid"
[138] "138. The Father"
[139] "139. Shutter Island"
[140] "140. All About Eve"
[141] "141. There Will Be Blood"
[142] "142. Jurassic Park"
[143] "143. Casino"
[144] "144. The Sixth Sense"
[145] "145. Ran"
[146] "146. Top Gun: Maverick"
[147] "147. No Country for Old Men"
[148] "148. The Thing"
[149] "149. Pan's Labyrinth"
[150] "150. Unforgiven"
[151] "151. A Beautiful Mind"
[152] "152. Kill Bill: Vol. 1"
[153] "153. The Treasure of the Sierra Madre"
[154] "154. Yojimbo"
[155] "155. Prisoners"
[156] "156. Finding Nemo"
[157] "157. The Great Escape"
[158] "158. Monty Python and the Holy Grail"
[159] "159. Howl's Moving Castle"
[160] "160. The Elephant Man"
[161] "161. Dial M for Murder"
[162] "162. Gone with the Wind"
[163] "163. Rashomon"
[164] "164. The Wild Robot"
[165] "165. Chinatown"
[166] "166. Klaus"
[167] "167. The Secret in Their Eyes"
[168] "168. Lock, Stock and Two Smoking Barrels"
[169] "169. V for Vendetta"
[170] "170. Inside Out"
[171] "171. Three Billboards Outside Ebbing, Missouri"
[172] "172. Trainspotting"
[173] "173. The Bridge on the River Kwai"
[174] "174. Raging Bull"
[175] "175. Catch Me If You Can"
[176] "176. Fargo"
[177] "177. Warrior"
[178] "178. Harry Potter and the Deathly Hallows: Part 2"
[179] "179. Gran Torino"
[180] "180. Million Dollar Baby"
[181] "181. Spider-Man: No Way Home"
[182] "182. My Neighbor Totoro"
[183] "183. Mad Max: Fury Road"
[184] "184. Ben-Hur"
[185] "185. Children of Heaven"
[186] "186. Barry Lyndon"
[187] "187. 12 Years a Slave"
[188] "188. Before Sunrise"
[189] "189. Blade Runner"
[190] "190. The Grand Budapest Hotel"
[191] "191. Dead Poets Society"
[192] "192. Hacksaw Ridge"
[193] "193. Gone Girl"
[194] "194. Memories of Murder"
[195] "195. In the Name of the Father"
[196] "196. Monsters, Inc."
[197] "197. Ratatouille"
[198] "198. The Gold Rush"
[199] "199. Wild Tales"
[200] "200. How to Train Your Dragon"
[201] "201. Sherlock Jr."
[202] "202. Jaws"
[203] "203. The Deer Hunter"
[204] "204. Mary and Max"
[205] "205. The General"
[206] "206. Ford v Ferrari"
[207] "207. The Wages of Fear"
[208] "208. On the Waterfront"
[209] "209. Mr. Smith Goes to Washington"
[210] "210. Wild Strawberries"
[211] "211. Maharaja"
[212] "212. Logan"
[213] "213. The Third Man"
[214] "214. Rocky"
[215] "215. Tokyo Story"
[216] "216. The Big Lebowski"
[217] "217. Spotlight"
[218] "218. The Seventh Seal"
[219] "219. The Terminator"
[220] "220. Room"
[221] "221. Pirates of the Caribbean: The Curse of the Black Pearl"
[222] "222. Hotel Rwanda"
[223] "223. La haine"
[224] "224. Platoon"
[225] "225. Demon Slayer: Kimetsu no Yaiba - Tsuzumi Mansion Arc"
[226] "226. Jai Bhim"
[227] "227. Before Sunset"
[228] "228. The Best Years of Our Lives"
[229] "229. The Exorcist"
[230] "230. The Passion of Joan of Arc"
[231] "231. The Wizard of Oz"
[232] "232. The Incredibles"
[233] "233. Rush"
[234] "234. The Sound of Music"
[235] "235. Hachi: A Dog's Tale"
[236] "236. Stand by Me"
[237] "237. Network"
[238] "238. My Father and My Son"
[239] "239. The Handmaiden"
[240] "240. The Iron Giant"
[241] "241. To Be or Not to Be"
[242] "242. The Battle of Algiers"
[243] "243. Into the Wild"
[244] "244. The Grapes of Wrath"
[245] "245. Groundhog Day"
[246] "246. The Help"
[247] "247. A Silent Voice: The Movie"
[248] "248. Amores Perros"
[249] "249. Rebecca"
[250] "250. A Man Escaped" Extract year
Extract runtime
Extract MPAA rating
Put the pieces together
Error in `tibble()`:
! Tibble columns must have compatible sizes.
• Size 250: Existing data.
• Size 245: Column `mpaa`.
ℹ Only values of size one are recycled.Error:
! object 'imdb_top_250' not foundScraping mpaas directly doesn’t catch the NA’s
There are 250 movies but only 245 MPAA ratings
Solution: scrape movies first, and then extract elements:
What do we do if scraping doesn’t work?
- Is there an API available? Can I access it as a student/researcher at no/low cost?
- IMDb: Yes, but cost prohibitive
- Is there a different scraping tool available?
- Yes, {RSelenium} is one that might work, but beyond the scope of this course
- Can I get the information I need from a different website?
- The Movie Database might work
- If I download the page, is more information available?
- In this case, yes, but might not always work
Ethics
Considerations
- We don’t want to scrape a website more than we need to - server hits
- For projects that involve intensive data-gathering:
- use an R script to read in the “raw” data, clean it, and save it to a tidy csv
- Read your “clean” data to your .qmd and proceed as usual - don’t rescrape every time to get your data!
- Again, check terms of service/use to see if a site allows you to scrape
- Even if scraping is legally allowed or website makes no comment, ask yourself whether people’s privacy is being respected with the scraping you are doing
Your turn:
In an R script:
Scrape the names, scores, and years of most popular TV shows on IMDB: www.imdb.com/chart/tvmeter
Create a data frame called
tvshowswith the variables:rank,title,stars,year,episodes,n_ratingsWrangle your resulting data so that all variable types are imported correctly
Use
write_csvto save your file. If time, read it into the20-scraping.qmdand make a graph